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ABSTRACT 

A taxonomy of programmatic tasks in an educational 
evaluation and facilitation coordination system vhich encompasses 
tasks appropriate tor evaluators, members of an evaluation unit, and 
members of facilitation and coordination units is proposed. In this 
comprehensive approach to the evaluation process, the taxonomy 
classifies and specifies tasks to effect evaluation, facilitation and 
coordination. It is suggested that such a conceptualization of the 
evaluation process differs from that of Ovens and Stuff lebean, in 
that the process becomes more rigorous and necessitates the 
identification and performance of additional tasks. (Author) 
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Statement of Focus 

The proposed taxonomy of prograonatlc tasks Iri^n educational 
evaluation and facilitation coordination system encompasses 
tasks appropriate for evaluators » T&embers of an evaluation 
unit» and members of facilitation and coordination units* 

In this comprehensive approach to the evaluation process, the 
taxonomy classifies and specifies tasks to effect evaluation, 
facilitation and coordination. 

Such a conceptualization of the evaluation process differs 
from that of Owens and Stufflebeam. In our vievi the process 
becomes more rigorous and necessitates the Identification and 
performance of additional tasks. For example, the development 
of managerial plans. 
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A TAXONDMY OF PROGRA^ttlATIC TASKS IN AN EDUCATIONAL EVALUATION 
FACILITATION AND COORDINATION SYSTEM (REVISED)* 

by Alan R. Coller 



OVERVIEW 



The raison d^etre for the Cooperative Educational Research Laboratory, 
Incorporated (CERJ^I) was established as that of inventing new solutions for the 
operating problens of educational institutions. As a means of accomplishing 
this purpose I CERLI early concentrated its energies upon the much neglected, but 
nevertheless important, area of educational '^middleman'* role development. One of 
I the roles which CERLI initially focused upon was the role of the educational 
evaluator , CERLI 's interest in developing the evaluator role was heightened by 
(1) the realization that evaluation was to serve an increasingly major role in 
educational decision*making, (2) the dearth of research in this area, and (3) 
estimates which predicted a severe shor^ of educational evaluators in the 
near future. Thus, CERLI 's focvs on the educational evaluator represented an 
attempt to forestall wh^t appeared to be an impending crisia aituation. 

In time, CERLI established the Evaluator Development Prog ram* -a program 
designed to define tasks and roles for evaluators. The staff of this program 
believed that the evaluator role differed depending upon the institutional setting 
and upon the malor question asked of the evaluation. Thus, we find Bates, Buser, 
Ellis, and Rice indicating that: 



^^The overall objective of CERLI 'a Evaluator Development Program 
is to enhance effective evaluation processes, procedures, and 
practices in elet&enfary, secondary, and higher education through 
the institutionalization of evaluator roles within differen- 
tiated settings." 

CBBLX's Evaluator Task Development Project (ETD Project) represented 
the first step toward the development of the educational evaluator role. *l)is 
project was designed specifically to: (1) identify and select evaluative tasks 

appropriate for performance In differentiated educational eet tings, (2) identify 
the akills and competencies needed for the performance of selected evaluative 
tasks, (3) devise materials and processes for developing these skills and com- 
petencies, and (4) apply the materials and processes on a trial basic. Unfor- 
tunately, these objectives were not all reached, ror a number of reasons not 



*Aa earlier paper did not contain the overview and background sections. Some 
minor changes in the body of the paper have also been made, 

^Batet, D,, Buaer, R, L., Bllia, J,, and Rice, D., Evaluator Development 

Program." Cooperative Educational Research Laboratory, Incorporated, #10, 1967. 
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uncotnxDon to fledgling organizations » objectives 1 and 2 of the LTD Project were 
never satisfactorily cotztpleted. Instead, another group at CERuI begsn work on the 
3rd and 4th objectives. 

During its tenure, the staff at CEKLI^s ETD Project developed a list 
of about forty "evaluator tasks" and their associated "skills." CERLI staff^ 
also conducted an evaluation of these evaluator tasks and skills by soliciting 
the reactions of various professionals. In general, the consultants Indicated 
that there was some Inconsistency In the degree of specificity with which the 
evaluator tasks were described and that there was some duplication of tasks. More 
crucial, however, was the reactions of the consultants to the conceptual framework 
for the tasks; they suggested that the tasks be organized Into a Ilmiued number of 
major task categories according to a theoretical framework or taxonomy. Shortly 
after the receipt of the consultants reactions the £XD Project was Indefinitely 
discontinued, that Is, until this year. 

In the meantime, Thomas Owens^ at Ohio State Unlvers:lty wrestled with 
this very same problem and developed a solution, as his recent article "Suggested 
Tasks and Roles of Evaluation Specialists In Education" would seem to Indicate. 

A comparison of the two approaches- -CERLI's and Ovens ' --should prove Instructive. 



BACKGROUND: COMPARING TUO PROJECTS 



Owens^ approach to the classification Issue was quite different from 
that employed by CERLI'a ETD Project staff. Owens pre-selected a class If icatory 
schema*, refining and extending Daniel Stufflebeam's^ set of evaluation process 
(design) categories. Using a taxonomy as a starting point, Ovens proceeded to 
generate evaluator tasks. This procedure was reversed by CERLI's staff. They 
first generated evaluator tasks and then attempted to classify those tasks accord- 
ing to the principle of "common elements." 



As we Indicated earlier, the ETD Project staff assumed (quite correctly) 
that the specific tasks that an evaluator performs Is dependent. In pare, upon 
the specific type of evaluation to be undertaken. They assumed that projects 
having different evaluation goals (questions) e.g., professional personnel eval- 
uation, educational prograam evaluation, etc., would require the educational eval- 
uator to perform essentially different evaluator tasks. Tne development: of a 



^Suaaarv of Consultant Reactions to the CERLI Evaluator Pevelonment PropTtim with 
St^ff RecowmendaTl^M for Fu^re PrograaTpevelopme nt . Cooperative Educational 
Research Laboratory, Incorporatedi #12, 196P. 

^Owans, T. R., "Suggested Tasks and Roles of Evaluation Specialists In Iducatlon." 
Educational Technoloav . 1969, IX , p. 4-10. 

^tufflebeam, D. L», ^^Torvard a Science of Educational EvaUation," Bducstlonal 
Technologj, 1968, VIll, p. 5*12. 
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questlonnalre as against the development of an achievement test* for example* 

Undue concern for this factor led to the duplication of tasks and detracted from 
efforts to develop evaluator tasks which adequately reflected the evaluation 
process for even one of the evaluation goals with which the ETD Project staff 
were concerned. Owens avoided these problems oa three counts. He avoided dupli* 
eating tasks by (1) attending to only one type of evaluation proJect--the evalua** 
tion of educational programs, and (2) describing the tasks in general rather than 
specific terms* He also appears to have Included tasks which reflect the evalua- 
tion process since his taxonomy is based upon an evaluation process model. 

Actually, the adequacy by which his tasks reflect the evaluation process depends 
upon the degree to which his taxonomy adequately portrays the evaluation process, 
(We will have more to say about this Isiiue later,) 

In general, Owens bad a broader range of tasks while the ETD Pxo^ect 
staff had a broader range of ’’evaluator” tasks. This must sound strange but can 
easily be clarified, Owens developed tasks for an ’’evaluation unit” consisting of 
an evaluation director, an evaluation coordinator, a surveillance specialist, an 
instruments specialist, a data-collectlon specialist, a data processing specialist, 
and a reporting specialist, CERLI's staff developed tasks for ’’evaluators in 
differentiated settings” Thus, CERL7 focused on ’’evaluating” and thereby 
restricted their range of tasks. On the other hand, Owens' dependency upon 
Stufflebeam’s conceptualization of the evaluation process served to restrict his 
range of evaluator tasks. The ETD Project staff maintained an eclectic approach 
to the e/aluatlon process, incorporating in their task descriptions of the evalu- 
ation strategies of a number of theorlticlans who hold divergent or, at least, 
non- Isomorphic evaluation views. 



BACKGROUND: THE CURRENT PROJECT 

CERLI's Interest in the evaluator development project stensaed from 
their determination to clrcijavert a serious manpower shortage which Clark and 
Hopkins^ had predicted would soon occur, that is, if no rrmedlal steps were taken. 
The shortage never fully materialized despite legislated requirements for recipi- 
ents of government funds to evaluate. Clear needs for evaluator services were 
seldom translated into demands and school systems used available personnel, 
trained or not, to undertake evaluation projects. Indeed, Hopkins^ has recently 
indicated that updated versions of the manpower report may not even contain a 
category for the evaluator. After a careful examination of the issues it was 
suggested that CERLl should abandon, at least temporarily, their attempts to 
develop the evaluator role, and concentrate Instead on developing programs to 
train school personnel to perform certain evaluative tasks. However, CBRLI was 
also informed that it would not be enough to train school personnel to undertake 
evaluation projects, their activities must be supported, coordinated, and 



^Clark, D, L, and Hopkins, J., ’’Roles for Researchers, Developers, aud Dlssemlna 
tors.” CRP Project X-002, 1966. 

^Telephone conversation with John Hopkins, March 1969, 
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facilitated by other personnel. Toward this end a possible model for the facili- 
tation and coordination of educational evaluation was provided. 

Briefly, it was suggested that school personnel should be trained over 
a period of time to undertake a variety of evaluation projects and that their 
efforts should be facilitated and coordinated by trained evaluators and specialists 
in facilitation and coordination roles. Specifically, it was recommended that 
states should financially support several teams of evaluation and area specialists 
whose responsibility it would be to research certain evaluation-related areas, 
collect, develop, and disseminate evaluation materials, train school and evaluator* 
type personnel, and perform site-visits. With some support by the state, districts 
should combine their resources to hire a team of evaluation, facilitation, and 
coordination speclallbts who would have research, developm.^nt , diffusion, adoption, 
training, facilitation, coordination, and evaluative roles* Two roles should be 
developed at the district level; the Evaluation Facilitator and the Evaluation 
Coordinator. The Facilitator \n>uld function to persuade school personnel to 
undertake evaluation, to facilitate their activities, to support their activities, 
and to perform certain evaluative tasks. The major goal of the Facilitator would 
be to increase meaningfully and permanently the quality and usage of evaluation. 

The Evaluation Coordinator assists the Facilitator in several functions and, 
additionally, functions to coordinate evaluation activities within the individual 
school, within the district, and between districts. The major goal of the Eval- 
uation Coordinator would be to eliminate or reduce needless duplication of efforts 
and to Increase the general igability of findings by standardizing evaluation 
objectives, plans, end techniques over cooperating schools and districts. At the 
school level, it was recommended that educators should be selected to assume the 
role of an Evaluation Intermediary, and act as laison between the local school 
personnel and the Faclllta^.or and the Coordinator. Finally, it was suggested 
that parents trained as evaluator aides should be recruited to assist educators 
in their evaluation efl^orts. 



THE CURRENT PROJECT 

CBRLi's early attempt to develop evaluator tasks and Owens' later more 
sophisticated approach both represent significant contributions to this area. Our 
attempt at developing a classificatory schema for taskr^ in a "hypothetical" evalu- 
ation facilitation and coordination system is but a modest step forward in compari- 
son to their ground-breaking efforts. Indeed, we intend only to impose a new 
frame of reference upon previously initiated work and add to an already impressive 
and imposing list of tasks. We suppose that even more tasks could be added to 
our lists and fully expect that others will do so. Indeed, tasks related to 
systems analysis in education--planning-prograaaiing-budgeting systems, for example 
--have not yet been included, but should be. As far as ve are concerned the 
current project should not be looked upon as a final version. The tasks remain 
to be empirically tested against the real world. 

The impetus to engage in a task development project derives mainly from 
efforts to provide a more comprehensive base for efforts to develop roles in 
evaluation-related areas of education. Another reason for this project is related 
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to our Attempts to develop an evaluation facilitation and coordination syateto. 

Such a system cannot adequately be described until tasks have been defined. During 
the prccese of describing the system we will be defining and refining the roles 
of the participants of the system. In turn, the role definitions will act as a 
check against the lists of tasks. We could , for example, have unassigned tasks, 
or we could find that we Inadvertantly forgot to include certain tasks. 

The project differs in several ways from that of its predecessors’. 

Some of the differences seem irrel?vsnt-'*style differences, for example, are 
unimportant. Other differences, the focus, for Instance, is an important aspect 
to discuss. It is our intention to define tasks which would be appropriate for 
the participants in a system which includes evaluation, but also includes the 
facilitation and coordination of evaluation efforts. Our focus affects the scope 
of the project. This project includes tasks appropriate for evaluators, members 
of an evaluation unit, and members of facilitation and coordination units. Thus, 
the scope of this project is much broader than earlier attempts. 

Stufflebeam’s conceptualization of the evaluation process, with some 
refinement, serves as the basis for Oirans’ taxonomy. Differences between Owens' 
taxonomy and ours result from four sources. As already indicated, we are con-> 
cerned with a r ootp rehens ive evaluation, facilitation and coordination system** 
Owens was concerned only with an evaluation unit. Secondly, we have attempted 
to be more explicit. Thirdly, our conceptualization of the evaluation process 
differs from that of Owens and Stuff lebeam. We have made an attempt to envision 
a more rigorous process and have, therefore, included additional categories**the 
development of managerial plans, for example. And, fourthly, our conception of 
the evaluators’ responsibilities seems to differ from that of Owens. We envision 
the evaluator as being responsible for more than descript ion--he must also make 
Judgments • 



A TAXDMOHT OF PROGRAMMATIC TASKS 

A proposed taxonomy (classification scheme) fur tasks in an hypothetical 
educational evaluation facilitation and coordination system consists of the sixteen 
programmatic tasks as found in Table I. 
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TABLE I 



A Taxonotoy of Progranmatic Tasks in an Educational 
Evaluation Facilitation and Coordination System 



1. Developing supportive climates for evaluation 

2. Focusing an evaluation: I Selecting decision situations 

3. Focusing an Evaluation; II Selecting evaluation strate^^ies and developing 
evaluation plans 

4r Organising the evaluation and developing managerial plans 

5. Assessing, modifying and selecting evaluation and managerial plans 

6. Selecting, modifying or developing data-gathering techni<7ues 

7. Collecting data 

8. Data processing: £ Preparing "raw'^ data 

9. Data processing: II Treating data (analysic) 

10. Interpreting and Judging outcomes 

11. Reporting outcomes 

12. Information processing and dissemination 

13. Decision making: the utilisation of information 

14. Performing activities and research related to evaluation and to the develop- 
ment, diffusion and adoption of evaluation 

15. Administering and coordinating activities in an evaluation and coordination 
system, 

16. Providing facilitation and coordination ser\'ices 
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1 . DEVELOPING SUPPORTIVE CLI M ATES FOR EVALUATION 

There are two major conceptions o£ evaluation: (1) the traditional 

evaluation approach which focuses upon marking, grading, crediting, and the 
attainment of behavioral objectives through specific transactions; and (2) the 
more modem or current evaluation approach which Is Intent on providing the 
decision-maker with the kinds of Information he requires in order that he may 
arrive at a "rational'* decision about the "real" world. In this neoteric evalu- 
ation approach, "Anything goes... any reasonable means to an end."? 

4 

Despite this amaalng trend In evaluation theory, evaluation specialists 
and many of the members of the educational community (EH) have not appreciably 
Improved their working relationships that traditionally have jeopardized whatever 
mutual efforts have been undertaken. Since most educators are required to grade 
their students, they will evaluate or measure student outcomes. In general, 
however, educators resist or reject the use of evaluation for other purposes. 
They--or so It seems--have no desire to Initiate formal evaluation efforts, tra- 
ditional or otherwise, or to take part In such efforts. 

Many evaluators have arbitrarily assumed that the educators' professional 
Insecurity generates this uncooperative attitude. However, some evaluators' 
Insensate approach could catalyze negative reactions. In actuality, the educators' 
disillusionment with traditional evaluation practices may well account for such 
negativism and hostility. 

For eKample, StakeS has suggested that '^Today's educator may rely little 
on formal evaluation because Its answers have seldom been answers to questions he 
Is asking." (p. 523) Egon Guba^ concurs: "The traditional methods of evaluation 

have failed educators In their attempts to assess the Inmact of Innovations In 
operating systems." (p. 29) 

One cannot easily avoid the Implications that Stake and Guba draw: 
Evaluators and evaluation efforts are resisted by the educational community because 
the "oay-off" they 1have received to date has been totally Inadequate. Indeed, 
Guba^Oargues that traditional evaluation efforts have hurt more than have helped 
the educator. 

Efforts to relieve this Impossible situation by creating evaluation 
models which are more acceptable to educators have met, so far, with only partial 



'Which Is not to be confused with sloppiness. The current approach Is to use both 
objective and subjective measures* but to injure that t^ e subjective data has 
some objective base. 

^Stake, R. B. , "The Countenance of Educational Evaluation." Teachers College 
Record, 1967, 68, p. S23-S40. 

A 

'Gub«, E. G., ’*fbe Failure of Educational Evaluation." Educational Techno l oitv. 
1969, IX, p. 29-38. 

O 
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auccess* Even reasonable expectations that recently-enacted legislation vould 
rekindle the educators^ latent Interest in evaluation have been ext inguished-- 
predicted demand situations simply have not materiallaed. It would seem that 
most educators either do not know about the dramatic changes which have occurred 
in evaluation strategies or do nor fully comprehend the meaning of such changes. 

This communications problem must be solved, and participants in evalua- 
tion activities have the obligation to inform the educational community of the 
many advances in evaluation strategies and technlffues. Once the educational com- 
munity realizes the need for evaluation, the transition to a demand situation 
(though still great) is no longer the olympian task it now appears to be. 

Performance of the following tasks would Induce a supportive climate for 
evaluation: 

1.1 Developing alternative strategies based upon particular 
characteristics of select members or segments of the 
educational milisru in order to modify reactions towards 
evaluation 

1.2 Assessing the existing attitudes of selected ntembers of 
the educational milieu toward evaluation*#^^ 

1.3 Becoming thoroughly aware of the individual's or group's 
unique characteristics# 

1.4 Establishing trust and rapport with all members of the 
educational milieu, but particularly with pertici«^>«nt8 
of an evaluation effort#^ 

1.5 Identifying available neoteric evaluation strategies and 
techniques 

1.6 Kaking select members of the educational milieu aware of 
the discrepancies between the observed and the intended# 

1.7 Demonstrating, in other ways, the efficacy of evaluation 



llHePceforth, this symbol (*) will be used to represent a coriespondence of some 
degree between task descriptions within this paper and tasks as described by 
Owens. (See Owens, T.R., "Suggested Tasks and Roles of Evaluation Specialists 
in Education." Educational Technology . 1968, VllI, .p. 4-10), 

Henceforth, this symbol ^#) will be used to represent a correspondence of some 
degree between task descriptions within this paper and tasks as described by 
Bates, St al (See Bates, D., Buser, R.L», Ellis, J., and Rice, D., "The Evalua 
tor Developiaent Program." Cooperative Educational Research Laboratory, Inc., 
#10, 1967). 




11 



-9- 



1 »8 Idea^lfylng and reducing any Inhibitions toward evaluation^ 

1 »9 Instilling within the educational conmunlty an awareness of 
the need for evaluation# 

1*10 Instilling within the educational cora&unlty a demand for 
evaluation 

1 * 11 Reinforcing positive attitudes toward evaluation 

1«12 Stating clearly the purposes of an evaluation and the role 
of the evaluator* 

1 > 13 Indicating clearlyi when appropriate » the types of information 
that will result from an evaluation and to whom it will be 
tpade available 

1 >14 Establishing clearly operational procedures and mutual rules 
of "etiquette*^ with participants of an evaluation effort 

1 >15 Maintaining open communication channels among the participants 
of an evaluation effort 

I * 16 Thanking the participants of an evaluation effort for their 
cooperation 

1.17 Crediting! when appropriate» salected participants for their 
cooperation and aid in bringing an evaluation effort to a 
conclusion 

1»16 Providing! when appropriate, feedback related to the outcomes 
of an evaluation effort to participants of the evaluation 

1>19 Inviting and encouraging select members of the educational 
milieu to participate actively in evaluation efforts* 

1 >20 Supporting the efforts of persons Initiating and/or engaging 
in evaluation efforts 

1.21 Consulting with and supporting the efforts of professionals 
and para-professionals who are attempting to develop a 
supportive climate for evaluation. 



2. gOCUSIWG AH EVALUATIOKi I SELECTING DECISION SITUATIOHS 



As a nilei educators will not themselves seek to initiate formal eval- 
uation efforts unless (1) they are fully cognisant of the potential or current 
existence of a "problem" that saust or should be resolved; and (2) they are 
convinced that there is a need to systematically collect additional data and/or 
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Infonsation bearing on the problem. They will» of course, continue to apply 
grades and initiate evaluation efforts as a response to ''institutional press" 
described by Taylor^^ as "societal and professional pressures," i.e., charges 
from the school board or legislation requiring evaluation. 

With the possible exception of an evaluation specifically designed to 
reveal the existence of unrealized problems, evaluation planning cannot, under 
normal circumstances, meaningfully begin unless a need--a problem--has been noted 
and until educators have identified, evaluated, and selected specific "decision 
situations" as the foci of the evaluation effort. Smith^3 defined decision 
situations as "Any situation in which a Judgment must be made about a course of 
action*" The functioning textbook committee has entered into a decision situation, 
for example. Having been alerted by a colleague of the need to replace worn texts, 
they must now decide to reorder the same or to order a newer text. How they will 
arrive at a decision should concern us all. Will they rely solely upon propaganda, 
authority, tradition, or intuition? Or, will they systematically collect data 
and/or information about the two texts? The latter course of action certainly 
would be preferable. 

Not every educator is sufficiently experienced or skilled to recognize 
the existence of a problem that requires the decision-maker to develop a solution. 
Others, dreading the decision-making process itself, choose to ignore untoward 
indicators. Still others may respond inflexibly to a problem that is only par- 
tially understood. Thus, in addition to assisting in the identification, eval- 
uation, and selection of decision situations, evaluation specialists also must 
assume responsibility for training educators to be alert to educational needs-- 
to become more sensitive to "decision stimuli," i.e., those stimuli (worn textbooks) 
which direct the attention of the decision-maker, as well as significant others, 
to a decision situation. 

Tasks in this category include: 

2 . 1 Training select members of the educational milieu to become 
sensitive to decision stimuli (including "institutional presses") 

2.2 Responding to decision stimuli (including "institutional presses") 

2.3, Reinforcing those individuals sensitive to decision stimKii 

2.4 Identifying decision situations of interest and importance Co 
select members of the educational milieu 



^^Taylor, P.A., "A Theoretical Evaluation Kodel." Educacionsl and Psychological 
Measurement , 1967, 27, p. 305-321. 

^^Smith, G.R., "An Analysis of Research on Decision Situations and Processes." 
Paper presented at the Meeting of the Anetjcan Educational Research Association, 
February 1967, p» 3. 
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2*5 Determining the desirability of collecting evaluative information 
for each of the identified decision situations 

2,6 Identifying the major level(s) of decision-making and of decision- 
makers to be served* 

2>7 Identifying the decision«»making process as it operates in a given 
sotting# 

2>8 Determining meaningfulnesa and feasibility of collecting evalua- 
tive information for each of the identified decision situations 

2.9 Establishing priorities for the collection of evaluative infor- 
mat ion 

2.10 Selecting tentative , desirable » meaningful » and feasible decision 

situations for evaluation 

( 

3. Focusing AN EVALUATIpy; II SELECTING EVALUATION STRATEGIES AND DEVELOPING 
EVALUATION VLANS 



respite their obvious kinship^ there is no denying that educationel ) 
evaluation significantly differs from its pedagogically-orlented counterpart-- 
educational research. Each tends to ong>ha8i£e divergent educational concerns, 
for example. It is the intention of all edcM:ational evaluation to provide decision- 
makers (either present or future) with "practical" information related to the 
"worth" cf educational experiences, outcomes, and artifacts. Most of the time 
the evaluation will be concerned with describing or Judging 'Vhat" it is that is 
happening. At other times, the evaluation will be concerned with 'Vhy" a thing 
happens . 

Recently, Stake^^ suggested that evaluators "Have a fundamental choice: 
to be scientific, to generalise, to evaluate to find out why ; or to be descriptive, 
to be delimited, and to evaluate to rind out what ," (p, 41) 

In contrast, one finds that it is not always the intention of educational 
research to provide outcomes %fhlch have practical educational value; some educa- 
tional research (as it should be) is purely "theoretical," Furthennore, educational 
research is only a^ildly Interested in describing "what" is happening; Its major 
concern is to discover and confirm the causes and effects--the 'Sirhya"--of educa- 
tional experiences and outcomes. Of the two, evaluation is the more flexible and 
readily adapts, in its ascientiflc fora, to the natuial chaos of ongoing action 
programs. Basically a "laboratory" science, educational research is more restricted 
than evaluation and requires highly controlled conditions. 



^^Stake, R,B., "Ceneralizahility of Program Evaluation: The Need for Limits," 

Educatiopiil Product Report , 1969, 2, p. 39-41, 
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What kind of evaluation strategy should the evaluator utilize? Should he 
employ vhat Campbell and Stanley^^ have called ^’quasl-experlmental" research 
designs, or should he use the more traditional ''experimental" design? As a matter 
of fact, the evaluator is in a position to choose "pre<»experimental" designs. The 
choice, according to Stake^^, "Depends on how much and in what directions the 
findings are expected to generalize, to be relevant to programs other than the 
one observed." 

Another aspect of evaluation strategy concerns the categories of infot-* 
nation which should be gathered as part of the evaluation. Evaluation designs 
developed by Stake^7 and by Stuf flebeam^^ provide us with some answers to this 
problem. 



After an evaluation strategy has been selected, an evaluation plan must 
be developed. One must distinguish between, evaluation plans and managerial plans. 
E\^aluation plans are cor arned with both general design issues and the detailed 
accounting of the proposed content--the 'Vhat"-*«of the evaluation. Managerial 
plans also are concerned with specifics*-the detailed "how" of the evaluation. 

Tasks associated with this category are: 

3.1 Establishing criteria for decision*making regarding expected 
outcomes# 

3.2 Projecting the decision situations to be served in terms of 
their locus, focus, criticality, timing, and con^osition of 
alternatives* 

3.3 Becoming knovledgable about relevant aspects of settings, 
conditions, and/or contexts within which the evaluation will 
occur# 

3.4 Projecting the decision situations to be served in terms of 
political, social, institutional, and situational constraints 

3.5 Defining policies and limits within which the evaluation must 
operate 

3.6 Making explicit and clarifying project assumptions* 



^^Campbell, O.T. and Stanley, J.C., "Experimental and Quasi-^experimental Designs 
for Research on Teaching." In Cage, N.L. (Ed), Handbook of Research on Teaching . 
Chicago: Rand McKally & Co., 1963. 

^^Stake, R.B., lo<*. . cit . 

^^Stake, R.E., Countenance of Educational Evaluation." p. 523-540. 

^^Stufflebeaa, D.L., '^Toward a Science of Educational Evaluation," Educational 
Tcchnoioav . 1968, VIII, p. 5-12. 
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3«7 Requesting pertinent Information from surveillance specialists 
and retrieval centers 

3>8 Reviewing research literature concerning similar projects In 

order to: (1) verify assun^tlons » (2) uncover sources of possi- 

ble Incidental gains or unwanted side effectSi and (3) make as 
uniform as possible the use of tests and testing procedures*i^ 

3*9 Visiting pertinent persons eud places to acquire updated infor- 
mation 

3> 10 Developing the rationale and objectives for the decisions situa- 
tions 

3.11 Identifying or formulating the basic question and/or hypothesis 
of the evaluation# 

3.12 Establishing premises which will guide the evaluation* 

3 «13 Determining If the evaluation goals are formative, summatlve, 
or both 

3.14 Determining the level of generalization for the evaluation 

3.15 Identifying, when appropriate, the pre-experlmental , experi- 
mental, or quasl-experlmental research designs to be used In 
the evaluation# 

3.16 Identifying available evaluation strategies 

3.17 Selecting an appropriate evaluation strategy# 

3. 18 Utilizing, when necessary, the services of evaluation specialists 
to select evaluation strategies and develop evaluation plans 

3.19 Developing, If necessary, evaluation strategies# 

3*20 Identifying segments of the educational coosnunlty to be alfected 
by the Intended transactions and outcomes 

3*21 Detailing, when appropriate. Intended antecedents, transactions, 
and outcomes# 

3.22 Re-structurlng, when necessary. Intentions (objectives) into 
observable (measurable or descrlbable) outcomes* 

3.23 Listing, If appropriate, the potential existence of contingencies 
between antecedents, transactions, and outcomes 

3*24 Listing, if appropriate, the potential congruence between Intents 
and observations ^ 
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3.25 Making explicit and clarifying stand.irds for use in the Judgment 
of alternatives* 

3.26 Identifying and clarifying judgments required of the evitluator 

3t27 Identifyixig segments of the educatior.al milieu from vhlcH 
judgt&ents will be collected 

3. 28 Estimating, when appropriate, which data-gsthering techniques 
will be utilised in the collection of observables 

3.29 Identifying the sample# 

3y39 Relating the project or program to other efforts or experiences 

of others who have coped with similar or related areas, and showing 
how the project utilizes, builds upon, extends, revises, or adapts 
to existing knowledge# 

3.31 Coordinating efforts to develop evaluation plans 

3.32 Smoothing the efforts of others to develop evaluation plans 



4. ORGANIZING IKE EVALUATION AMD DEVELOPING MANAGERIAL FLAMS 



The recent incursion of network*based managerial systems into the social 
sciences (see the work of Desmond Cook.^^ for example) now makes it meaningful to 
create specialized plana--to distinguish between evaluation and manageria!). plans. 
Evaluation plans are designed, partially, to focus the evaluation--to sharply 
define what will be done to answer what basic questions. In contrast, marisgerial 
plans are concerned mainly with the systematic organization of evaluation com- . 
ponent8--a detailed description concerned with (1) how the evaluation will take 
place, step by step, and (2) what resources (money, manpower, facilities, mschlnes) 
will he required over what period of time* It is a cortplex step whereby a multitudie 
of divergent components are arranged into a coordinated program or project. Also, 
during this stage it is often appropriate, at the very least, to develop plans for 
the (1) selection, modification, or development of datn*gathei ing techniqu^is; (2) 
sampling of populations; (3) collecting of data (ircludlng procedures and data 
formats); (4) preparation of data; (5) treatment of data; (6) interpretation and 
Judgment of outcomes; (7) reporting of outcomes; (8) storage and dissemination of 
reports; and (9) utilization of information by decision-makers. Of course, not 
every evaluation effort will be concerned with developing every one of these nine 
plans, but even that must be determined. 



Maxtagemeot plana or systems baaed on the concepts of network planning 
and critical path analysis are known by a vide variety of abbreviated names. For 



19 

Cook’. D.j **PrograQ BvAlustlon and Review Tecl)dlque: Applications in Eduction. 

08-12024. Cooperative Research . Honoaranh Ko. 17. 1966>'. 
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examplej there are PERT (Progtao Evalxi'etioa add Review Technique) and CPM (Critical 
Path Method) procedures^ Such systems can allow for the planuing, schedulings 
and controlling of complex interrelated evaluation efforts and of the resources 
(times moneys manpower. facilitieSs machines) required to execute those activities. 
Archibald and Villoria^^ have written an excellent technical book on management 
systems. 



Tasks appropriate for this function are: 

4. 1 Reviewing and becoming familiar with the objectives of the 
evaluation effort 

4.2 Identifying those program and/or project components which must 
be completed in order to achieve the evaluation objectives 

4.3 Reviewing and becoming familiar with issues of criticality and 
timing* 

4.4 Identifying 'hnilestones^' and developing a planning structure for 
the program or project. 

4.5 Developing planning structures for individual projects or com- 
ponents 

4. 6 Developing plans for selecting or developing data-gathering 
techniques 

4.7 Developing sampling plans 

4.8 Developing plans for coding data 

4.9 Developing plans for collectiv^g data# 

4. 10 Developing plans for preparing raw data# 

4.11 Developing plans for treating data 

4. 12 Developing plans for interpreting and judging outcomes 

4.13 Developing plans for reporting data 

4.14 Developing plans for storing and disseminating information 

4. 15 Developing plans for infon&ation utilization 

4.16 Determining the dependency relationship existing among the 
identified project cooq^onents 



^^Archibaldi R. D. and Villoriai R. L., Hetwork-Based Management Systems 
(fBRI/C114> > Mew York: John Wiley A Sons, 1967. 
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4,17 Arranging the con^onents of the evaluation In a network according 
to a plan 

4> 18 Detemlng how long It will take to complete the program plan 

4,19 Determining the amount and type of resources required to complete 
the plan 

4 >20 Modifying the plans according to known restraints 
4 >21 Determining local resources available for the evaluation# 

4> 22 Defining the manpower and material needs for the evaluation*# 

4 >23 Constructing an evaluation budget for the project*# 

4 >24 Coordinating efforts to develop managerial plans 
4*25 Smoothing efforts to develop managerial plans 



5> ASSESSING^ MODIFYIMG^ AND SKLECTING EVALUATION AND MANAGERIAL PLANS 

As evaluation and managerial plans are being developed, they should be 
systematically assessed and a final assessment should be made before they are 
submitted for funding* Plans assessments should Include stylistic criteria, l>e>, 
general cooaunlcablllty > They also should include content criteria, l>e>, 
legality and practicality > In a relatively recent article, Caldwell^^ described 
a plana assessment framework and suggested criteria by which plans may be a8sessed> 

Tasks which should be Included with this factor are: 

5> 1 Reviewing evaluation and managerial plans for communication 
value, l>e>, are the plans clear and seductive 

5>2 Determining If the scope of the evaluation has been stated 
eiq>llcltly 

5>3 Determining when necessary. If the format of the proposed plans 
Is appropriate for the receiving agency# 

5*4 Learning the project rationale, objectives, and operational 
procedures* 

5>5 Determining the audiences, the declslon*makers to be served, 
and the nature of the Istplementlng agency* 

5*6 Using effectively subject area or technical specialists whenever 
necessary to review the evaluation and managerial plans# 



^^Caldwell, M> S>, ''An ^proach to the Assessment of Educational Planning >" 
Educational Technology , 1968, Vlll, p> 5-12* 
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5.7 Determining the relevance of the proposed evaluation plan to the 
identified decision 8ituation8@^^ 

5.8 Determining the relevance of the proposed managerial plan to the 
proposed evaluation plan 

5.9 Determining the legal status of the proposed evaluation and 
managerial plans relative to the context within which they are 
to be implemented^ 

5.10 Determining the congruence of the evaluation and managerial plans 
with the value systems of the context within which they are to 

be implementedt? 

5.11 Determining If the evaluation plan Is within the purview of the 
agency charged with the implementation^ 

5.12 Determining the compatibility cf the evaluation and managerial 
plans with the value systemCs)^ i.e,, purposes and goals of the 
implementing agencyC? 

5.13 Determining the impact of the evaluation and managerial plans on 
other components (sub-systeme) of the system and on the weights 
and interrelationships of these system elements^ 

5.14 Determining the practicality of the evaluation andkaanagerlal plans 
in terms of achieving its stated purposea fend-products)@ 

5.15 Determining the relative desirability of the evaluation and 
flianagerial plans (in comparison with other plans) In terms of 
the ratio of necessary inputs (costs) to expected outputs 
(effectivenesa)@ 

5.16 Consulting with clients in order to review and/or modify evalua* 
tion and managerial plans 

5.17 Hodifying the evaluation and managerial plans in terms of the 
outcomes of the assessment 

5.18 Coordinating assessment efforts 

5.19 Smoothing assessment efforts 

5.20 Submitting the proposed plans for approval and funding 



^^enceforth» this symbol i)9) will be used to represent a correspondence of some 
degree 8 between task descriptions within this paper and tasks as described by 
Caldwell (Ibid ,) 
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6. SELECTING. MODIFYING. OR DEVELOPING DATA-GATHERING TECHNIQUES 

Both the educational evaluator and the educational researcher share an 
Interest In so-called ''objective'* Information. In addition to "objective Infor- 
mation," however, the evaluator Is equally as Intrigued by "subjective Informa- 
tion." As Stake^3 has explained, "An evaluation of a school program should portray 
the merit and fault perceived by well-ldentlf led groups, systematically gathered 
and processed. Thus, Judgment data and descriptive data are both essential to 
the evaluation of educational programs." Thus, the evaluators' Interest In all 
forms of data (be they "soft" or "hard") portrays his commitment to describe fully 
the "countenance" of an educational experience. 

As one would predict, the evaluator^s sweeping data Interests are ^unply 
reflected In the eclectic assortment of data-gatherlng techniques that he habit- 
ually employs In his studies. These techniques range from standardized psycho- 
metric tests through unsystematized personal observances. For this reason, we 
have Inserted the term "techniques" for the term "Instruments." Since the eval- 
uator Is free to use direct personal observations as a means of gathering data, 
he can subjectively describe persons, places, things, and processes which could 
not otherwise be so easily described. He need not (though often does) rely upon 
replicable procedures. Often, the acceptance of the evaluator's description Is 
based upon conviction that the evaluator Is astute enough to understand the 
environment . 

To better describe outcomes, the evaluator recently has become Interested 
In "criterion-referenced" tests. A relevant bibliography Is presented by Baker^^ 

In a recent view. 

Tasks considered Important for selecting, modifying, or developing data- 
gatherlng tec'r.niques Include: 

6 . 1 Reviewing and becoming familiar with the Intents of an evaluation 
study 

6.2 Stating the purposes for which data-gatherlng techniques are to 
be used^ 

6.3 Stating explicitly the objectives for which data-gatherlng 
techniques are to be used# 

6.4 Utilizing, When necessary, the services of test and measurement 
specialists 

6.5 Reviewing resource materials related to similar projects to 
uncover suitable data-gatherlng techniques# 



^^Stake, R. E., "The Countenance of Educational Evaluation," p. 527 

^^Baker, R. L., 'Curriculum Evaluation," Review of Educational Research, 1969, 
39, p. 339-354 
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6^ Securing the oaintuining copies of data^gathering techniques# 

6.7 Utilizing, vhen necessary, the services of surveillance specialists 

6.8 Developing criteria for selecting the tnost suitable available 
date-gathering techniques^# 

6.9 Identifying pertinent techniques for which evidence indicates 
effectiveness# 

6.10 Selecting suitable data-gathering techniques 

6.11 Modifying, field-testing, and revising identified data-gathering 
techniques# 

6.12 Developing specifications for constructing data-gathering techniques 
if no existing data-gathering techniques are appropriate* 

6.13 Developing, pilot testing, and revising new data-gathering 
techniques*# 

6.14 Collecting, vhen appropriate, reliability and validity information# 

6.15 Collecting, vhen necessary, norevat: ;e information for use as 
standards 

6.1 6 Preparing administrative and scoring manuals for the newly modified 
or developed data-gathering techniques 

6.17 Training personnel in the administration of various data-gathering 
techniques 

6.18 Consulting with clients regarding available data-gathering 
techniques 

6.19 Advising clients regarding the development, validation, and 
nortaing of various data-gathering techniques 

6.20 Aiding select members of the educational nilreu in the application 
of sound tests and measurement policies, programs, and practices# 

6.21 Persuading others to cooperate in the development, validation, 
and nooning of specific data-gathering techniques 

6.22 Obtaining resources and resource personnel to facilitate the 
development, validation, and noroing of specific data-gathering 
techniques 

6.23 Coordinating efforts to select, develop, modify, validate, and/or 
norm data-gathering techniques 
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6.24 Smoothing the efforts of others to select^ modify, develop, 
validate, or norm data-gathering techniques 

6.25 Researching new data-gathering techniques 



7. COLLECTING DATA 

The implementation of evaluation plans often begins with the collection 
of data. This is frequently true even if a "pre-test" is not administered, for 
the evaluator is concerned with describing transactions as they take place. As 
a method, data collection is not independent of the techniques used to gather the 
data. The administration of some data-gathering techniques requires the examiner 
to be highly skilled and experienced. Indeed, the administration of some tech- 
niques requires great sensitivity--an "artistic" talent. But there are many other 
techniques which do not require such skill. For example, many techniques are 
daily administered by educators who are less well-trained than professional 
examiners. Because of this, one can anticipate that parents (trained as par<*- 
professional evaluation aides--a role not unlike the teacher's aide) might assist 
the educator during an evaluation by administering data-gathering techniques. 

Tasks related to the collection of data are: 

7 . 1 Specifying infomation needs clearly and concisely# 

7.2 Identifying information sources (populations and individuals) for 
the collection of data* 

7.3 Identifying information environments for collecting data 

7.4 Specifying methods to be used in collecting data* 

7.5 Specifying sampling procedures*# 

7.6 Specifying the schedule for data collection* 

7.7 Reviewing the saispling plan and schedule with relevant others 
for appropriateness and congruence with other ongoing programs 

7.8 Utilising, when necessary, the services of data collection 
specialists 

7.9 Preparing the sample population and relevant others for data 
collection 

7 . 10 Field-testing data collection methods 

7.11 Training personnel to collect and record data* 

7.12 Informing personnel of the rules of "etiquette" for collecting 

data 
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7 . 13 Adi^inlsterlng evaluative data*gatheriag techniques and 
recording the data* 

7 . 14 Reviewing and modifying, whet necessary, data collecting 
plans and methods 

7 . 15 Obtaining resources and resource personnel for data col- 
lection efforts 

7.16 Persuading others to cooperate in the collections and 
recording of data 

7 . 17 Coordinating efforts to collect data 

7 . 18 Smoothing the efforts of others to collect and record data 

7 . 1 9 Researching data collection techniques 

8. DATA PROCESSIWQt I PREPARING *'RAW» DATA 



In examining the evaluation process, three major types of processing 
activities --two categories of *'data^' processing and one category of ''information*' 
processing can be identified. By data processing we shall mean, (1) the sys- 
tematic transformation and organization (the preparation) of "raw'* data into 
forms suitable for the application of analytical operations and (2) the systematic 
treatment of transformed and organized data during the actual analytical process. 
This latter set of activities includes the use of computers and computer- related 
equipment. Information processing refers to those activities related to the 
preparation of already prepared, treated (^nalyze^/, interpreted. Judged, and 
reported data and outcoiaes for purposes of Issemination, 

Under ordinary evaluation conditions one seldom finds freshly collected 
data, that is, raw data or unprocessed responses, to be in a form suitable enough 
to be analyzed without first being specially prepared. The raw data processing 
function mainly entails (1) the conversion of unprocessed responses into scores 
or categories; (2) the transformation of scored responses into data storage cards 
or tabulation sheets; (3) a systematic organizing and re-organizing of coded 
responses into forms consistent with intended analytical operations; snd (4) the 
storage and retrieval of data. Tasks adjunctive to the above include the opera- 
tion of scoring machinery and computer- related equipment. 

Tasks which would fall within the domain of raw data processing include: 

8.1 Providing specifications for the scoring and/or classification 
of data 

8.2 Becoming familiar with the data, the intended analytic process, 
and available computer programs 
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8.3 Providing formats for coding data which are compatible with 
available coo^uter programs or analytic procedures* 

8.4 Training personnel in response interpretation 

8.5 Training personnel to operate mechanical scoring units 

8.6 Scoring and/or classifying responses obtained from the admin- 
istration of data-gathering techniques* 

8.7 Utilizing, when necessary, the services of professional scoring 
services 

8.8 Transcribing memorial or taped data when necessary 

8*9 Training personnel to operate machines related to the preparation 
of computer data cards 

8 . 10 Transferring raw data onto computer cards, tabulation sheets, 
or other data storage systems 

8.11 Utilizing, when necessary, the services of data processing 
assistants 

8.12 Providing for data storage, management and retrieval 

8.13 Utilizing, when necessary, the services of data processing 
specialists, and test and measurement specialists 

8.14 Informing relevant members of the educational milieu regarding 
the types of raw data available for heuristic purposes# 

8.15 Coordinating data processing activities* 

J.16 Dicseminating raw data upon the request of appropriate officials 

8.17 Consulting with clients regarding the preparation of raw data 

8.18 Obtaining resources and resource personnel for preparing raw data 

8.19 Smoothing the efforts of others to ptepare raw data 



9. DATA FROCESSniG: 11 TRBATIWG DATA (APALYSISl 

Fred Rerlinger^^ defines "analysis" as "The ordering, the breaking down 
of data into constituent parts in order to obtain answers to* •• {evaluation] 



^^erllnger, F. H. , Voundationa of Behavioral Research , Hew York: Holt, Rinehart 

and Winston, Inc*, 1965, p. 603. 
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quest Ions," He explains that, "The analysis of (evaluation] data, however does 
not in and of itself provide the answers to. «• [evaluation] questions. Interpre- 
tation of the data is still necessary. Thus, data analysis is a process during 
which prepared data are systematically treated for the purposes of description, 
interpretation, and the rendering of Judgments. 



During this process the evaluator or the statistical cpecialiet or the 
data prograaning specialist systematically manipulates prepared data according 
to operations dictated by statistical formulae and the "language" of the computer. 
Indeed, in the last decade computer science has progressed so rapidly that the 
accomplished statistical specialist is one who has become well acquainted with 
computer programaing--a marriage seemingly enhancing both disciplines. 



Tasks which can be associated with this function are; 



9.1 Reviewing the objectives of the evaluation 

9.2 Determining the level of sophistication required by decisloa- 
make rs 

9 .3 Reviewing the evaluation design actually employed 
Reviewing the sampling procedures actually employed 

9^.^ Determining the nature of the data collected 

Determining desired levels of statistical precision 

Reviewing the research literature for new statistical procedures 

9.8^ Utilizing, when necessary, the services of statistics! 
specialists 

9.9 Becoming familiar with available computer programs 
Selectii.g the analytical procedures* 

9.11 Using existing computer programs# 

j[ji Ij Writing new computer programs when necessary* 

9.13 Utilizing, when necessary, the services of program writers 

9.14 Designating a means for performing the anlysis* 

9.15 Developing a library of cookbook-like statistical forms and 
procedures 
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9> 16 Performing tVie statistical computations* 

9.17 Treating data to test hypothesis determining relationships 
and/or to answer basic questions 

9 .18 Obtaining resources and resource personnel to facilitate the 
analytical process 

9.19 Utilizing, when necessary, the services of computer and computer- 
related equipment operators 

9.20 Coordinating efforts to treat the data 

9.21 Smoothing the efforts of others to treat data 

9.22 Consulting with clients regarding statistical procedures 

9.23 Researching new analytical procedures 

9.24 Producing computational documentation when appropriated 

9.25 Advising clients regarding available computer programs 

9. 26 Consulting with clients regarding computer programming and 
operations 

9.27 Maintaining a library of computer programs and their descriptions 

9.28 Determining the types of computer programs which will be demanded 
in the future 

9 .29 Disseminating information pertaining to the acquisition and/or 
modification of computer programs 



10. IWTERPRETIMG AND JUDGINS OUTCOHES 



The evaluator who "interprets" the outcomes of an 
them for their meaning and implications. Interpretation as 
treated data and makes inferences regarding vaiious aspects 
The evaluator might, for exaasple, make inferences about the 
to exist between intentions and observations. According to 
are congruent if what was Intended actually happens. To be 
intended antecedents, transactions, and outcomes would have 
(This seldom happens* -and often should not.)" 



evaluation explores 
a process takes the 
of the evaluation, 
"congruence" as found 
Stake, '*The data... 
fully congruent the 
to come to pass. 



^^Stake, R# E., loc. cit. 
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The evaluator also may attempt to draw conclusions about 'VontlngencieR" 
existing among antcedents, transactions, and outcomes. Stake distinguishes 
between "logical contingencies" and "empirical contingencies": 

'Whenever intents are evaluated the contingency criterion is 
one of logic. To test che logic of an educational contingency 
the evaluators rely on previous experience^ perhaps, or 
research experience, with similar observables. No immediate 
observation of these variables, however, is necessary to test 
the strength of the contingencies among intents. 

'^Evaluation of observation contingencies depends on enpirical 
evidence. To say, 'this arithmetic class progressed rapidly 
because the teacher was somewhat, but not too sophicticated 
in ndthematics' demands empirical data, either from within 
the evaluation or from the research literature . 

Interpretation is but one concern of the evaluator; he must also indicate 
the perceived merits and shortcomings of whatever it was that he evaluated. Stake 
and Denny29 claim that: 

"Evaluation is not a search for cause and effect (as is 
research), an icrventory of present status, or a prediction 
of future success. It is something of all of these but 
only as they contribute to understanding substance, function, 
and worth." 

Striven^ also has a similar concept of evaluation: 

''Evaluation is itself a methodological activity which is 
essentially similar whether we are trying to evaluate coffee 
machines or teaching machines, plans for a house or plans for 
a curriculum. The activity consists simply in the gathering 
and combining of performance data with a weighted set of goal 
scales to yield either comparative or numerical ratings, and 
in the justification of (a) che dsta - gathering instruments, 

(b) the weightings, and (c) the selection of goals." 



^^Stake, R. B. and Denny, T., "Heeded Concepts and Techniques for Utllltlng more 
fully the Potential of Evaluation." In the Sixty-eighth Yearbook of the National 
Society for the Study of Education; Educational Evaluation: New Roles. Uew 

Means . Chicago: University of Chicago Press, p. 370. 

^^Scrlven, M, S., "The Methodology of Evaluation." In AERA Monograph Series on 
Curriculum Evaluation: 1. Perspectives on Curriculum Evaluation . Chicago: 

Rand McNally, 1967, p. 39*83. 
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Flnally^ the evaluator l8 urged to make recommendations. 

Tasks related to the interpretation and judgment of outcomes are: 

10.1 Reviewing the objectives of the evaluation 

10.2 Eecoming thoroughly familiar with the transactions of the 
evaluation; the basic questions asked; the criteria, model, 

^ rocedure, and techniques employed; and the data collected^ 

10.3 Verifying the statistical analysis and the procedures used 
to collect and process the data 

10.4 Comparing observed antecedents, transactions and outcomes with 
intended antecedentSi transactions, and outcomes# 

10.3 Describing points of congruence and incongxuencc# 

10.6 Making inferences about contingencies among transect ions and 
outcomes 

10.7 Interpreting the results of the evaluation program In terms 
of given criteria* 

10.8 Establishing relationships and distinguishing between those 
outcomes that result from treatment application r.nd those 
contingent upon antecedent oondltioos# 

10.9 Utlli2ing, when necessary, the setvlces of evaluation, 
statistical, and subject area specialists in order to 
interpret the data 

10.10 Determining if any unintended outcomes occurred# 

10.11 Identifying the ways In which segments of the educational 
community, i.e., the institution, its staff, students, and 
community, etc., were affected by the observed transactions 
and outcotnes 

10.12 Comparing generalisations from the literature and theory 
with those drawn from the evaluate sn outcones# 

10.13 Reviewing the Judgments required of the eveluator 

10.14 Obtaining Judgments concerning the outcomes of the evaluation 
from relevant members of the educational milieu 

10.15 Judging the 'Stforth*' of the outccfses of the evaluation 

10.16 Rendering Judgments regarding the w^rth of alternative 
strategies as employed in the evaluation# 
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10. L7 Rendering judgments as to the significance of the observed 

transactions for various segments of the educf^tional milieu# 

10.18 Judging if the unintentional outcomes are unwanted side 
effects or incident^al gains# 

10.]^.9 Rendering Judgments regarding the worth and relevance of 
data-gatheriog techniques as used in the evaluation# 

10.20 Rendering Judgments as to the overall quality of the eval- 
uation effort 

10.21 Recommendlnf.» if appropriate, future modifications for the 
data-gathering techniques 

10.22 Identifying and renonanending alternative strategies# 

10.23 Identifying and recommending procedures to control or 
reduce unwanted aide effects# 

10.24 Identifying and recommending procedures to control 
enhance incidental gains 

10.23 Providing counsel to relevant members of the educational 
milieu regarding the interpretations and implications of 
the Judgments rendered# 

10.26 Consulting with clients regarding interpretation and Judging 
outcomes 

10.2.7 Obtaining resources and resource personnel to facilitate 
the interpreting and Judging of the outcomes 

10.28 Coordinating efforts to interpret and Judge outcomes 

10.29 Smoothing efforts to interpret and Judge outcomes 



Reporting evaluation cutcomea requires an unusually high degree of 



technical skill and evaluation "savvy." Stake and Denny^l emphasize this point 



"Evaluation reports, unfortunately, usually tell little more 
than that the work proposed was coeapleted, that the complaints 
of the staff were Justified, and that there were greater 
differences within groups of students (or schools or rurric- 
uluat) than there were between the groups. Host evaluation 



11. RbFORTIWC OUTCOHES 
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reports give only the participants some notion of what occurred; 
the outsider gains little insight. Most formal reports avoid 
explicit subjective Judgments by insiders and outsiders as if 
they were evil, 

'Educators and laymen alike cannot non-vlsuallee and explain 
what is happening In our classrooms. Part of the reason for 
this falling is our inability to share perceptions and 
measurements. Part is out lack of motivation to share them. 

What should be told? What should be shared? Our needs are 
not only procedural; we need also a commitment to full and 
accurate reporting," 

The reporting function Is the fifth part of Stuff lebeam's^^ evaluation design. 
Concerned less with what should be told and shared, Stufflebeam stresses the 
procedural aspects of reporting; 

purpose of this part, the reporting part of a design, 
is to insure that decision-makers will have timely access 
to the information they need and that they will receive it 
in a manner and form vhl^h facilitates their use of the 
Informatlor." 

One must accept the fact that evaluation report writing Is a difficult, often 
unrewarding Job. 

Tasks which would be associated with reporting the outcomes of evalua- 
tion are: 

11.1 Becoming familiar with available means for reporting outcomes 

11.2 Researching new methods for reporting outcomes 

11.3 Specifying the format for evaluation reports 

11.4 Becoming avare of the professional understandings of those 
who are to receive whe evaluation results# 

11.5 Scheduling the reporting of outcomes 

11.6 Specifying means for reporting the outcomes of the evaluation 
to relevant audiences* 

11*7 Coordinating efforts to prepare a report 

11.3 Preparing a roport of the evaluation that will be under- 
standable to the public. It will serve and include 



^^Stufflebeam, D. L., loc. clt. 
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meanlngful terms » tables, charts, graphs, illustrations, 
and answers to the basic questions of the evaluation^ 

11.9 Describing the intended antecedents, transactions, aud 
outcomes 

11.10 Describing processes and procedures by which evaluative data 
were gathered and Judgments rendered# 

11.11 Describing the observed antecedents, transactions, and outcomes 

11.12 Describing criteria# 

11.13 Describing explicitly unintentional outcomes# 

11.14 Detailing rendered Judgments 

11.15 Reporting the limitations of the evaluation# 

11.16 Utilizing, when necessary, the services of reporting specialists 
in order to report the outcomes 

11.17 Obtaining resources and resource personnel to facilitate i:he 
reporting of outcomes 

11.18 Coordinating efforts to produce a report 

11.19 Preparing findings and recommendations to the decision-makers 
in an understandable manner*# 

11.20 Obtaining the decision-makers' reactions to the report 

11.21 Providing evaluation abstracts or summaries for presentation 
to specific groups* 

11*22 Packaging the outcomes of the evaluation for purposes of 
presentation to appropriate publics# 

11*23 Obtaining reactions to the presentations 

11.24 Consulting vrith clients regarding the reporting of outcomes 

11.25 Smoothing the efforts of others to prepare and produce reports 

11.26 Reporting subsequent modifications in transactions and their 
observed outcomes# 
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12. INFORMATION PROCESSING AND DISSEMINATION 



Barlier, "infonoation processing" was described as the act of processing 
already-prepared, treated, Interpreted, Judged, and reported data for purposes 
of dissemination. Information proceselng activities include reviewing evaluation 
literature, collecting evaluation reports, organizing and storing evaluative- 
based information, packaging evaluative Information and disseminating information 
pertaining to the field of evaluation. ''Data banks" also are included in this 
category. 

Oddly, there are no existing agencies, large or small, that have assumed 
the responsibility for processing and disseminating all forms of evaluative infor- 
mation. Some agencies have assumed the responsibility of processing and dissein- 
inatine specific types* of information. (Of note is the work of Fopham and 
Skager’®,) (Their plans involve collecting, storing, processing, and distributing 
objectives and evaluation measures.) And other agencies have disseminated general 
evaluation information for the non-professional evaluator, for example. The 
combined efforts of the Cooperative Educational Research Laboratory, Inc. (CERLl) 
and the Center for Instructional Research and Curriculum Evaluation (CIRCE) 
resulted in the dissemination of an "evaluation kit," containing pertinent eval- 
uation Information.^^ Clearly, some super-agency to assume the responsibility 
of processing and disseminating evaluation-related information should be estab- 
lished.* 

Guba^^ has suggested the development of, 

"A National Information Center for Education (NICE), wi.lrh 
would have as Its purpose organizing, processing, and report- 
ing evaluative information. It would develop and maintain 
rxchanieots that would serve these ends. The agenev would 
maintain records of the goals and organization of Federal 
and other funding progrmos in support of education, of the 
objectives and procedurer. of program elements within those 



■*-*Popham, W. J, and Skager, R, tf , , "Instructional Objectives Measurement System. 
Progress in Evaluation Study," Third Annual Report to the U. S. Office of 
Education, Center for the Study of Evaluation. Los Angeles: Graduate School 

of Education, University of California, 1968, p. 113-115. 

-'^Educational Products Information Exchange Institute, "Evaluation Kit: Tools 

and Techni<iues ." Educational Products Report, 1969, 2, Information Supplement 
#5 (whole). 

Cuba, E« G., "Confronting the Problems of ^Educational Evaluation: A Call for 

a CoLi^ortium of Relevant Agencies." A paper presented at an Invitational 
Conference on Educational Evaluation, Washington, D. C., 1967, p. 11, 13. 

*$ince this was written the Educational Testing Service has entered into the 
Educational Resources Information Center (ERIC) network and will dispense infor- 
tDation related to tests, measurement, and evaluation. 
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agencies, of the Information requirements of major audiences 
for education, and particularly the scope, sequence, timing, 
locus, focus, and criticality of decisions requiring evalua- 
tive Information* NICE would also be responsible for devel- 
oping operational Instruments for such data collection and 
treatment/' 

Cuba la Indeed anticipating a system which will be welcomed by all educators 
and especially by evaluators. However, until such an agency Is developed, groups 
of evaluators must take on some of the activities described below* 

Tasks which seem to be associated with this category are: 

12*1 Providing specifications for the coding and cataloguing 
of evaluative Information and processed data 

12*2 Training personnel In Infoi^tlon processing and dissemination 

12.3 Detailing the scope of information processing and dissemination 
activities t>erformed by the given agency 

12. A Identifying potential sources of evaluative information and 
processed dat«*. 

12.5 Requesting evaluation Information and processed data from 
likely sources 

12.6 Collecting evaluative Information and processed data 

12.7 Visiting pertinent persons and places to acquire updated 
evaluation information 

12.8 Utillrlng, if necessary, the services of surveillance and 
subject area specialists 

12.9 Conducting conferences regarding specific evaluation-related 
Issues of current Importance 

12>10 Requesting select members of the EM to prepare reviews of specific 
evaluation areas 

12.11 Coding and cataloguing evaluation Information and processed data 

12.12 Utilising the services of Information processing specialists 

12.13 Transferring evaluation information into information storage 
, systems 

12.14 Providing for Inforwhiion storage, management, and retrieval 
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12 « 15 In^lementlng and maintaining a data bank*# 

12«16 Becctnlng familiar with available information media methods 
for reporting information 

12> 17 Developing^ if necessary, new reporting styles 

12 . 18 Specifying the format for reporting evaluation information 

12> 19 Scheduling the reporting of evaluation information 

12 >20 Reporting evaluation information in a highly communicable 
fashion 

12>21 Providing evaluation abstracts or suamiaries 

12>22 Utilizing! if necessary, the services of reporting specialists 

12> 23 Identifying potential recipients of evaluative information and 
processed data 

12 >24 Informing relevant members of the educational milieu regarding 
the types of evaluation information which may be requested 

12>25 Communicating to potential users regarding the availability 
and use of the data bank# 

12 >26 Packaging the information for purposes of presentation to 
appropriate publics# 

12 >27 Disseminating evaluative Information and processed data 

12 >28 Consulting with clients regarding the retrieval of evaluative 
information and processed data 

12 >29 Coordinating evaluation information processing and dissemina- 
tion activities 

12 >30 Coordinating data bank activities 

12>31 Smoothing the efforts of others to acquire specific evaluation 
information and processed data 

12>32 Obtaining resources and resource personnel to process and 
disseminate evaluation ' 'iformation and processed data 

12 > 33 Obtaining reactions from relevant members of the educational 
milieu regarding the reporting of evaluation information 
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12,34 Surveying select members of the educational milieu regarding 
their evaluation Information needs 



12> 3S Determining future evaluation Information needs 



13- DECISION MAKING: THE UTILIZATIOy OF INFORMATION 



The evaluation process has now run Its full cycle. We began this process 



by modifying attitudes » that ls> by attempting to Instill within the educational 
community a demand to engage In formal evaluation projects. Then we trained 
educators to become sensitive to decision stimuli so that they might later Identify 
decision situations. Following this, evaluation strategies T'<rre selected and 
evaluation and managerial plans were developed • After the plans had been com- 
pleted, they underwent a final assessment and were modified If deemed necessary. 
Data-gatherlng techniques were selected or developed and data was collected. 
Finally; the raw data was systematically prepared, treated. Interpreted, judged, 
reported, and dlsaeoilnated. ' ' 

/ Now a iec Is Ion-maker has Information, and for all Intents and purposes 

/ the evaluation will be terminated. The ‘'feedback'* of Information to a declslon- 
/ maker--the goal of an evaluation s^£fort--has been accomplished. But, the ways In 
' which the decision-makers employ the Information they receive Is as crucial to 
the evalua\:lon process as any of the preceding steps. If dec Is loo-makers do not 
use formally gathered Information (even to the extent of Informing themselves 
about *Srfhafc" Is happening), then the raison d^etre for evaluation efforts has 
vanished* <Vn evaluation effort makes no sense if Its product --Informat Ion- -Is 
not utilised. 



In this respect, Wllhelms36 suggests that: 

"Regardless of whether the evaluation Is formal or Informal-- 
and equally regardless of whether It Is 'good' or 'sensitive* 
or *adequate*--lt has one thing In common with every other 
system of feedback: When It has blended Into the background 

system of purposes and values and policies, controls the 
next step . This is simply a fact of life; all our decisions 
are conditional by perceptions of how we are doing In terms 
of what we hope to do." 

That "next step" can be excruciatingly painful especially If the 



decision-maker Is unfamiliar with declslon-maklng strategies. This Is a likely 
occurrence according to Guba^^^ who claims that: 



^^llhelms, F. T., "Evaluation as Feedback." In Wilhelms, P. T. (Ed) Evaluation 
as Feedback and Guide * Washington, D. C.; Association for Supervision and 
Curriculum Development, KEA, 1967| p. 3* 

^^Guba, E. G*, "The Failure of Educational Evaluation," p* 33. 
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"At present I no adequate knowledge of decision processes and 
associated infonoation requirements relative to educational 
programs exist. Nor is there any ongoing program Co provide 
this knowledge." 

Not too long ago, Scriven^^ advised the evaluator to assume the responsi- 
bility for developing objectives. Similar advice would not be inappropriate now. 
EvaluAtors lshour4,in all due haste, seek ways to develop programs designed to 
better understand the decision processes. This is a must! If the decision- 
maker for any reason cannot utilise evaluative information, he will not be Inclined 
to engage in or support future evaluations. It would be most tragic if efforts 
e;.'pended to create a supportive climate for evaluation come to naught. 

Tasks relative to the decision-making process are; 

13.1 Becoming familiar with the contents of an evaluation report 

13.2 Assessing the perspicacity of the evaluation report 

13.3 Using effectively select members of the educational milieu: 
to assess the perspicacity of the evaluation report 

13. 4 Determining the nature of the Institutions involved in the 
decision solution# 

13.5 Revietfing the major level (s) of decision-making to be served* 

13 .6 Reviewing the decislon-maklog process as It operates in a 
given setting*# 

13.7 Developing taxonomies of educational decisions 
13 .6 Identifying decision-making criteria 

13.9 Selecting decislo.i-makirvg criteria 

13.10 IdenMfying decision-making models 

13.11 Developing, If necessary, a decision-making model 

13.12 Selecting a decision-B^aking model 

13 . 13 Identifying and proposing alternative decision solutions 

13 . 14 Assessing the decision solution for its relevance, legality, 
congruence, relatedness, compatability , impact, practicality, 
and relative desirability 



^^Scrivcn, M, W. , loc. cit. 
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13*15 Utilizing effectively select members of the educational milieu 
to assess the proposed decision solution 

13*16 Selecting desirable and feasible decision solutions 

13*17 Reinforcing those individuals who produce "rational" 
decision solutions 

13 * IB Reinforcing those individuals who produce "creative and 
feasible" decision solutions 

13*19 Training select members of the educational milieu in 
decision-making strategies 

13*20 Inviting and encouraging select members of the educational 
milieu to participate actively in decision-making processes 

13*21 Coordinating the decision-making process 

13.22 Providing directives, guidelines, and/or other needed 

assistance to decision-makers for purposes of improving 
the usefulness of transmitted data in the decision-making 
process# 

13*23 Smoothing the efforts of decision-makers to derive and select 
decision solutions 

13 . 24 Obtaining reactions to the decision solutions from relevant 
members of the educational milieu 

13*25 Assisting the decision-maker in formulating new questions 
for future evaluation based upon the original evaluation 
findings#* 

13*26 Usii*g evaluation findings as the basis for diccussion in 
teacher in-service training sessions* 

13*27 Conducting sensitivity sessions to induce selected members 
of the educational miHeu to laake use of the evaluative 
information in decision solutions 

13*26 Suggesting to decision-makers techniques by which they 
themselves can evaluate and modify behavior* 
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14. PERFORMING ACT1V1TI£S AND RESEARCH RELATED TO EVALUATION AND TO T HE 
DEVELOPMENT. DIFFUSION^ Am> ADOPTION OF EVALUAtlON 



Research specifically related to evaluation is vital to the success of 
evaluation as a discipline « It is imperative that new evaluation techniques and 
strategies be developed. Referring to this latter point, Guba^^ indicates that: 

**These facts lead me to the conclusion that some new evalua*^ 
tion strategy free of these [see below] defects is required 
before evaluation as a science can make its next major strides. 

Until we have developed a theory which permits evaluation to 
occur continuously (and to provide continuous feedback of 
relevant data) , to be free of the constraints imposed by 
conventional experimental design theory, to be open to alter » 
ations and refinements in the treatment during the evaluation 
period, and to yield results valid in the sept i c world of th e 
classroom as well as in the antiseptic world of the laboratory, 
we will make little progress.** 

More research is needed to determine how well our available strategies 
and techniques fare under close scrutiny. Evaluation»related variables of all 
kinds must be examined and normative informat ion»-standards-*must be collected 
and processed* But, if evaluation as a discipline is ever to be adopted by the 
school community, more activity and additional research related to development, 
diffusion, and adoption activities (as described by Clark and Guba^O ar|<] as 
amplified by Stuff lebeam and Westerlund^^) must be performed. If neoteric eval- 
uation strategies can be classified as innovations, they become subject to the 
procedures related to and necessary for change in education. 

Tasks related to the performance of research related to evalu.=)tion and 
activities related to the development, diffaaion and adoption of evaluation are: 

14.1 Defining explicitly the meanings of evaluation-based terns 

14.4 Developing and testing evaluation models* 

14.3 Performing research related to new methodological techniques* 



^^Guba, E. G., '*Confronting the Problems of Educational Evaluation: A Call for 

a Consortium of Relevant Agencies,'* p. 5« 

^^Clark, D. L. and Cuba, G. , *'An Examination of Potential Change Roles in 
Education." A paper read at a Seminar of Innovation in Planning School Cur- 
ricula, October 1965. 

^^Stuf flebeam, D. L. and Westerlund, S. R. , "llie Evaluation of Context, Input, 
Process and Product in Elementary and Secondary Education." U. S. Office of 
Education, Bureau of Elementary and Secondary Education, February 1967. 
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14.4 Drafting plans for constructing idealized evaluation strategies 
and techniques for usage in select settings oJ the educational 
milieu 

14.5 Constructing idealized evaluation strategies and techniques 
for usage in select settings of the educational milieu 

14*6 Integrating the components of the strategies and techniques 
into operating systems for usage in select settings of the 
educational milieu 

14.7 Developing procedures for creating widespread awareness of the 
neoteric evaluation strategies and techniques 

14.8 Developing situations by which individuals can examine and 
assess operating qualities of the neoteric evaluation strategies 
and techniques 

14.9 Researching procedures for the training of local personnel to 
manage » operate | service, and utilize neoteric evaluation 
strategies and techniques 

14*10 Developing situations for the trial use of the evaluation 
strategies and techniques 

14*11 Modifying the neoteric evaluation strategies and techniques 

to fit the particular circumstances* of the adopting Institution 

14.12 Performing studies to assure the assimilation of the evaluation 
strategies and techniques by the adopting Institution 

14*13 Conducting longitudinal studies to determine effects of 
specific variables over time* 

14. 14 Performing experimental research on some of the substantive areas 
being evaluated* 

14.15 Collecting standards of all kinds 

14*16 Developing a taxonomy for standards 

14. 17 Performing case studies or other type of research to learn more 
about the nature of children Involved in programs being evaluated# 

14.18 Conducting simulation studies and predictive studies* 

14*19 Conducting surveys related to educational needs, and uses and 
abuses of evaluation 

14.20 Determining the applicability of various data*gatherlng techniques 
for special populations 
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14*21 Comparing alternative strategies and techniques for instilling 
in select members of the educational milieu an awareness of a 
need for evaluation 

14.22 Comparing alternative strategies for instilling in select 
members of the educational milieu a demand for evaluation 

14.23 Determing attitudes toward evaluation and readiness for change 

14.24 Coordinating research and activities relating to the development, 
diffusion and adoption of evaluation 

14.25 Coordinating evaluation based research in general 

14.26 Smoothing general research activities and activities related to 
development, diffusion and adoption of evaluation 



15. ADMINISTERING AKD COORDINATING ACTIVITIES IN AN EVALUATION FACILITATIO N 
AND COORDINATION SYSTEM 



All organizational entities require some personnel to direct and super- 
vise the activities of the staff. Though not always directly pertinent to the 
evaluation process* these are vital functions. In the real world the evaluation 
probably would not be initiated without the aid of the administrative staff. 

Activities to be included within the framework of administraflcn-type 

tasks are: 

15.1 Stating explicitly the broad purposes of an evaluation and/or 
facilitation entity^ 

15.2 Developing specific policies and general guidelines for the 
operation of an evaluation and/or facilitation entity 

15.3 Identifying and assessing alternative objectives as possible 
goals for an evaluation and/or facilitation entity# 

15.4 Defining criteria for selecting objectives for an evaluation 
and/or facilitation entity# 

15.5 Selecting and assigning priorities to objectives for an evalua- 
tion and/or facilitation entity# 

15.6 Stimulating and assisting in periodic evaluation, reflection, 
and revision of purposes and/or objectives# 

15.7 Defining staff and resource requirement for operating sn eval- 
uation and/or facilitation entity* 

15.6 Developing plans to meet staff and resource requirements* 
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I5> 9 Developing Job descriptions 

15.10 Constructing, securing, and managing budgets* 

15.11 Developing policies and procedures for the selection, assignment, 
retention, dismissal, promotion, and In-service growth of 
personnel# 

15.12 Establishing criteria for evaluating the on-the-Job performance 
of personnel 

1 5.13 Developing policies and techniques for evaluating on-the-job 
performance of personnel# 

15.14 Reviewing all evaluation designs. Instruments, and reports before 
they are used or released for distribution* 

1 5.15 Identifying sources, l.e., foundations or agencies, which have 
Indicated an. Interest in supporting programs or projects 
similar in kind to the submitting evaluation and/or facilltatlou 
entity# 

15.16 Kaklng an Informal contact with the agency to which the proposal 
will be submitted# 

15.17 Developing an overall managerial plan for an evaluation and/or 
facilitation entity 

15.18 Supervising the training, research, facilitation and coordination 
services performed by staff members* 

15.19 Arranging for in-service training of the staff* 

15.20 Organising the tasks within the entity In order to utilise 
the unique talents of each member* 

15.21 Maintaining concitlona conducive to high morale and Job 
efflc lency 

15.22 Arranging for an Independent evaluation of the activities of an 
evaluation and/or facilitation entity* 

15.23 Coordinating evaluation, facilitation and/or coordinating 
activities within the evaluating group 

15.24 Coordinating evaluation, facilitation and/or coordinating 
activities within the school 

15.25 Coordinating evaluation, facilitation and/or coordinating 
activities within the district 
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15.26 Coordinating evaluation^ facilitation and/or coordinating 
activities between districts 

15.27 Coordinating evaluation, facilitation and/or coordinating 
activities within a cooperative multi-district unit 

15.28 Coordinating evaluation, facilitation and/or coordinating 
activities between cooperative multi-district units 

15.29 Coordinating evaluation, facilitation and/or coordinating 
activities within the state 

15.30 Smoothing the administrating and coordinating efforts of others 



16, PROVIDING FACaiTATION AW) COORDINATION SERVICES 

It would seem logical to consider evaluation a **service-oriented^' pro- 
fession not unlike that of law and medicine whote practitioners provide Informs- I 

tion and formulate decisions for their clients. In one respect, however, eval- | 

uation differs: the professional evaluator's clientele consists of "amateur^' ^ 

evaluators who know all about or at least express opinions about educational \ 

practices, institutions, problems, etc, | 

Nevertheless, in this age of specialization, some select evaluators I 

should function as facilitators of the decision-making process. Professional 
(highly qualified) evaluators not only should ^'evaluate" but they should facilitate 
and coordinate evaluation efforts that dynamize the decision-making process. 

An Educational Evaluation Facilitation and Coordination System would 
provide facilitative services for evaluation and decision-making efforts. To 
accomplish its mission, the system would be designed to increase the quality and 1 

usSLge of evaluation strategies and techniques in educational systems by providing 
services to professional, para-professional, tetoporary and non-professional eval- 



Tasks associated with these services include: 

16.1 Surveying the training needs of select members of the 
educational milieu# 

16.2 Developing instructional objectives, plans, aids, and 
materials for training# 

16.3 Training select members of thr> educational milieu in 

tasks associdted with their roles ' ' 

16.4 Training select members of the educational milieu in 

tasks related to evaluation facilitation and/or coordinatic.i 



uators » 
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16» 5 Obtaining from select members of the educational milieu 
reactions to training 

16.6 Surveying the service needs of select members of the 
educational milieu 

16t7 Facilitating, generally, the efforts of select members of 
the educational milieu to undertake evaluations 

1 6.8 Assisting select members of the educational milieu to 
develop objectives# 

1 6»9 Reducing impediments to evaluations 

16.1 0 Developing systems of support and reinforcement to those 
individuals undertaking evaluative efforts 

16*11 Obtaining from select members of the educational milieu 
reactions to facilitative efforts 

16.12 Facilitating, generally, the efforts of select members 
of the educational milieu to develop solutions for 
operatl.ig problems 

16.13 Obtaining resources and resource personnel for the facili* 
tation of evaluation efforts, the coordination of evalua- 
tion efforts and for decision-making efforts 
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